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ABSTRACT 

This report reviews the outcomes of a forum convened 
to examine policy and practice issues surrounding the annual 
evaluation of effectiveness of programs and services for students 
with disabilities. At the forum, alternative approaches being used by 
states to implement program evaluation were discussed, along with 
issues in gathering accountability data and the role of the federal 
government. The meeting resulted in identifying 11 characteristics of 
a statewide evaluation system that addresses the education of 
students with disabilities: clear def ini t i ons /purposes , feasible, 
flexible, capitalizes on prior work, is results-oriented with the 
goal of improving instructional practice, reflects consensus on 
outcomes, is inclusive of all students, contains incentives and 
supports, meets multiple levels of need and use, involves 
stakeholders, addresses accountability dilemmas, and is linked to 
reforms for all students. Recommendations are offered for action by 
the Office of Special Education Programs and state education 
agencies. Appendixes contain a participant list, an agenda, and a 
paper by Kenneth Olsen titled "Have We Made Progress in Fifteen Years 
of Evaluating the Effectiveness of Special Education Programs?," 
which contains 72 references. The Olsen paper identifies five strands 
of change that have implications for special education program 
evaluation such as the changes in emphasis from program improvement 
to accountability and from simple to complex conceptual models. 
(JDD) 
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A FORUM ON STATEWIDE EVALUATION OF PROGRAMS AND SERVICES 
FOR STUDENTS WITH DISABILITIES 



I, PURPOSE AND ORGANIZATION OF THE POLICY FORUM 



A Background and Purpose of the Forum 

The Individuals With Disabilities Education Act (IDEA) requires each State Plan to 
"provide for procedures for evaluation at least annually of the effectiveness of programs of 
meeting the educational needs of children with disabilities (including evaluation of 
individualized education programs),./* (Section 1413(a)(ll). In addition, Section 435(b)(4) 
of the General Education Provisions Act (GEPA) calls for the State to evaluate the 
effectiveness of covered programs (including the IDEA) at least every th^ee years. 

During the past 20 years, educators have struggled with ways to carry out these 
requirements (e.g., to evaluate the effectiveness of educational services for children and 
youth with disabilities) State education agencies have varied considerably in their 
interpretation of federal program evaluation requirements. For the most part, state 
education agencies have utilized monitoring compliance as a method of evaluating program 
effectiveness* 

Within the events of the education reform movement, there has been a shift from 
documenting the process of educating students to demonstrating positive outcomes. 
Legislators, governors, State and local boards of education, and the public are all demanding 
increased program accountability. 

On August 2-3, 1993, Project FORUM convened a forum to examine policy and practice 
issues surrounding the annual evaluation of program effectiveness. A follow-up policy forum 
held a year later in August, 1994 was intended to expand and build on discussions of the 
earlier forum. Specifically, the policy 1994 forum was convened to carry out the following 
objectives: 

o To identify components/parameters of a statewide system to evaluate 
programs and services for students with disabilities. 

o To consider the challenges and opportunities in 

the planning and implementation of such a system. 

o To consider existing and needed resources to assist SEAs in planning and 
implementing statewide evaluation systems 
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o To identify policy changes and resources needed at the federal, state, and 
local levels that can facilitate the planning and implementation of statewide 
evaluation systems/procedures. 



& Preparation for the Forum 

Project FORUM staff worked with the Office of Special Education Programs (OSEP) 
personnel to identify 16 participants for the forum (Appendix A). Participants included state 
directors of special education, college/university representatives, Regional Resource Center 
staff, a representative of a parent/advocate organization, local school district administrators, 
and private organization representatives. 

Dr. Judy Schrag, from Project FORUM, prepared a paper to serve as background to this 
forum, Evaluating the Impact and Effectiveness of Special Education Based on Program and 
Student Outcomes. This paper included a review of selected reports, documents, and articles 
pertaining to statewide evaluation of programs and services for students with disabilities. 
In addition, several other program evaluation papers were sent to participants prior to the 
August forum or were included in forum packets: 

o Brauen, M., O'Reilly, F. & Moore, M. (1994). Issues and options in outcome- 
based accountability for students with disabilities. College Park, MD: 
University of Maryland. 

o Project FORUM (1993). A forum to examine policy and practice issues 
surrounding the annual evaluation of program effectiveness. Alexandria, 
Virginia: National Association of State Directors of Special Education. 

o Olsen, K. (1994). Have we made progress in fifteen years of evaluating the 
effectiveness of special education programs? Lexington, Kentucky: Mid-South 
Regional Resource Center Interdisciplinary Human Development Institute. 

o Olsen, ¥L ( 1994). Bibliography on evaluating effectiveness of services to students 
with disabilities. Lexington, Kentucky: Mid-South Regional Resource Center 
Interdisciplinary Human Development Institute. 

o Ysseldyke, J., et. al. (May, 1994). Students with disabilities & educational 
standards: Recommendations for policy & practice. NCEO Policy Directions, 
Number 2. Minneapolis, Minnesota: University of MinnesotOy National Center 
on Educational Outcomes. 
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o Ysseldyke, J,, et, aL (May, 1994), Guidelines for inclusion of students with 
disabilities in large-scale assessments. NCEO Policy Directions, Number 1. 
Minneapolis, Minnesota: University of Minnesota, National 
Center on Educational Outcomes. 

o Ysseldyke, JL, Thurlow, M, & Geene, K. (1994), Implementation of alternative 
methods for making educational accountability decisions for students with 
disabilities. Synthesis Report 6. Minneapolis, MN: University of Minnesota, 
National Center on Educational Outcomes, 

Extensive work has been carried out by previous and existing projects funded by OSEP, 
as well as individual researchers. This background material was important to assist forum 
participants in not "plowing ground already covered" in previous forums/ meetings and other 
work, 

C Process of the Meeting 

The forum on program evaluation was held on August 30-31, 1994 at the Ritz Carlton 
Hotel, Pentagon City, Arlington, Virginia, The agenda (Appendix B) began with welcoming 
remarks from Joy Hicks, Director, Project FORUM, and Dr Lou Danielson, Director, 
Division of Innovation and Development-OSEP, Dr, Thomas Hehir, Director, OSEP, 
provided a context for the program evaluation forum. He indicated that considerable gains 
have been made for students with disabilities and their families during the past two decades. 
It is important that we focus on the results of these efforts. Within the federal government, 
there is an emphasis on developing indicators of success for various federal programs, 
including the IDEA. Dr, Hehir also indicated that we will need to look at the total system. 
If we are going to have a greater focus on program and student outcomes, we will have to 
give up on some things and strengthen others. There is a need to move away from an 
exclusive look at compliance with the procedural requirements of IDEA, and to focus on 
"outcomes" or results of our educational system on students with disabilities. 

The first morning of the forum proceeded with a panel presentation on alternative 
approaches being used by states to implement program evaluation. Dr. Jim Ysseldyke 
discussed several current evaluation alternatives: use of child count, administration of norm- 
referenced tests, focus on results or outcomes, aggregation of IEP objectives or goals 
attained, secondary analysis of extant data, and accreditation programs. Dr. Ysseldyke also 
discussed the following issues encountered by state personnel in their efforts to gather 
accountability data: 

The attitude that evaluation is not: important or a priority. 
Lack of commitment to get people to collect data. 
Complex process of agreeing on outcomes and standards. 
Determination of the source of responsibility for student learning. 
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Issues of inclusion in assessment. 
How to report information and what to report. 
Determination of the role of the SEA. 
Use of sanctions and rewards* 

Determination of the utility-is it worth it to shift resources from program to 

data collection? 

Fear. 

Local control. 

Lack of agreement on how to measure complex skills* 
Procedures for dealing with students in non-graded classrooms. 
Absence of utility, (when data is collected, no one uses it). 

Dr. Ysseldyke reported on a meeting held by the National Center for Educational Outcomes 
last September. Participants in that meeting indicated that a heavy federal involvement will 
be needed to help overcome the barrier of collecting data and to implement outcome 
evaluation rather than focusing on monitoring program processes. Participants at this 
meeting and others have indicated that they need data on the following to demonstrate that 
education works for studenxs with disabilities: 

L Input 

1. Resources (e.g., money spent per pupil, staffing ratios, 

instructional time). 

2. Student characteristics (e.g., percentage of first graders with 

kindergarten experience, student mobility, number of 
limited English speaking students, Chapter 1 students, 
number on free/reduced priced meals). 

77. Process 

1. Opportunity to learn. 

2. Inclusion of students with disabilities in general education. 

3. Teacher expectations for individual pupil performance. 

4. Extent to which IEFs translate into instruction. 

///. Outcomes 

1. Academic and functional skills. 

2. Valued social and emotional outcomes. 

3. Generalization of school learning to everyday life. 

4. Student and parent satisfaction. 

5. Independent living. 

6. Community participation. 
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7. Extent to which the "product" of schools meets the needs of the 
labor market* 

Dr. Ysseldyke also discussed several recommended practices: create incentives for 
people to gather data; make certain the data collection process makes sense to those who 
will implement and those who will be impacted (stakeholder involvement and buy-in); do 
not invent new data collection activities; use an outcomes-based approach (e.g., NCEO self- 
study guide); train those who will gather data; and make sure data collected are used. 

Dr. Mary Ann Lachat reported on program evaluation efforts in New Hampshire. 
The Center for Resource Management has been assisting the New Hampshire Department 
of Education to build state and local capacity to improve student results through the use of 
a wide array of data. Dr. Lachat described New Hampshire's new statewide standards-based 
assessment program which has been designed to include all students, including those with 
disabilities. A key purpose of the program is to promote higher standards of learning and 
increased academic achievement for all students. Statewide assessment data will be 
desegregated for students with disabilities to examine program effectiveness for this 
population. Dr. Lachat indicated that the NH Department of Education is also providing 
support to help local school districts develop the information system capacity necessary to 
support accountability, site-based evaluation, and program improvement. CRM has 
developed a software program that creates an integrated database, linking student 
performance, demographic data, and programmatic data. Data can be aggregated at the 
state level and desegregated at the local level for planning. The stale agency has provided 
support to this effort through the "tate Agency/Federal Evaluation Studies (SAFES) 
Program and the New Hampshire School Improvement Program. The program is being 
installed at several demonstration sites-training is provided to principals, special educators, 
and data entry personnel. 

Ms. Nancy LaCount presented the Kentucky Instructional Results and Information 
System which is based upon academic expectations. This system has multiple evaluations, 
including portfolio assessment in math and reading. Kentucky is beginning to collect 
massive data with a new technology system. Student questionnaires are being used including 
questions about homework. Other data collected includes extent of time missed from 
school, if students are using a higher type of math, type of post school plans, non-cognitive 
information, etc. Patterns of performance within a classroom can target technical assistance 
needed. Data and information gathered can guide local schools into transformation planning. 
The Kentucky assessment program is beginning to look at information about students with 
disabilities and has developed statewide alternative portfolio systems for assessing students 
with more severe cognitive disabilities. This sub-system is integrated in the overall system.. 
Along with an outcome focus, Kentucky continues to use monitoring and conducts special 
studies on specific issues. Data and information being gathered is being utilized for 
planning. A final point was made about the need for dedicated staff nnd professional 
development in order to implement statewide program evaluation procedures. 
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Dr. Richard Baldwin, Director of the Michigan Office of Special Education, reported 
on their outcome-based system. He stated that Michigan has outcomes established for each 
of its twelve disabilities areas, at preschool, third, sixth, ninth and graduation levels. The 
outcomes were developed with extensive input from parents, teachers, administrators, people 
with disabilities, and university personnel Michigan is also undergoing extensive training 
of teachers, administrators, and parents in how to us** the outcomes (assessing students and 
interpreting data). Michigan uses Part D training funds, as well as state-initiated project 
dollars for this massive training component. The outcome-based program in Michigan is not 
mandated; use of outcomes is at the discretion of school administrators and teachers. 
Lacking a mandate, the system is not used as extensively as Dr. Baldwin and others would 
hope. Finally, he pointed out that the outcome project fits nicely with reform initiatives in 
the Michigan education system. 

Following lunch, Dr. Ken Olsen, Mid-South Regional Resource Center, presented a 
"strawman" to serve as a focus of discussion and reaction v by the forum participants 
(Appendix C). A proposed statewide system of program evaluation with the following eight 
components was presented: 

1. Consensus on philosophy, purpose, and definitions 

2. Consensus on student outcomes and conceptual framework 

3. Ongoing data on student outcomes 

4. Training and technical assistance for local programs 

5. Ongoing data on educational resources, practices, and results 

6. SEA policy studies 

7. Data based decision making 

8. Reporting to stakeholders 

Dr. Olsen proposed ten foundations for statewide program evaluation: 

1. Standards or guidelines for evaluating effectiveness must be based 

primarily on best practices in evaluation rather than on legal 
obligations. 

2. The requirement to evaluate effectiveness of all IEPs is a charge to 

look at the effects not the document. 

3. An effective educational program is one that achieves valued outcomes 

and protects individual and group rights while using resourcos 
and activities that are directly related to those outcomes and 
rights. 

4. Evaluation involves comparing an actual state of affairs to a desired 

state of affairs. 
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5. Our goal is to evaluate the effectiveness of educational supports and 

services to students with disabilities not to evaluate "special 
education", 

6. The function of evaluation is to provide information to reduce 

uncertainty in decision making. 

7. All evaluation is formative in relation to some level of the 

organization. 

8. Evaluation does not have to be external to be useful, proper, or 

accurate. 

9. The potential of evaluation data to serve program improvement 

purposes is compromised when those same data are used for 
accountability purposes by a supervising agency. 

10. SEAs have unique and essential roles in modeling and supporting 

evaluation for program improvement in addition to their roles 
in the area of accountability. 

Following group discussion of these perspectives and the proposed statewide system 
of program evaluation, participants within three small groups began to identify suggested 
components and parameters for a statewide program evaluation system, as well as to identify 
issues for consideration in the planning and implementing of a statewide program evaluation 
system. 

The second day of the program evaluation forum included a discussion of 
challenges/opportunities in the implementation of program evaluation. In addition, the 
forum participants identified needed policy changes and assistance at the federal, state, and 
local levels to support the planning and implementation of state evaluation systems and 
procedures. 

II. OUTCOMES OF THE MEETING 

Project FORUM staff utilized laptop computers to record the large group discussions 
during the program evaluation forum. Flip chart paper and acetate overlays were utilized 
by the small groups to record discussions. Following is a summary of responses within 
eleven major categories to the question: 

What are the characteristics of a statewide evaluation system that address the 
education of students with disabilities? 
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Clear Definitions/Purposes 

o Defines terms within the system of evaluation for clear communication. 

o Articulates key evaluation questions. Contains a mechanism for further 
developing and reviewing evaluation questions. 

o Presents a set of belies developed by stakeholder consensus. 

o Focuses on outcomes of education and student attained of valued outcomes. 

o Is logically congruent among components. 

o Is fair with respect to gender, race, culture, and disability status. 

Feasible 

o Uses assessment that minimizes the burden on students, that does not detract 
from instruction, and that enhances and promotes learning. 

o Can be operationalized efficiently. 

o Utilizes appropriate technology to be integrated with extant databases so that 
it is aligned with data on all children. 

o Is cost effective. 

o Has reasonable standards based on student improvements. 
Flexible 

o Uses varied quantitative and qualitative data sources to get varied 
perspectives. 

o Accommodates various data collection methods that focus on outcomes of 
education. 

o Includes multiple options for assessment of student attainment of valued 
outcomes. 

o Accommodates the changing nature of schooling and evolving educational 
practices. 
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Capitalizes on Prior Work 

The development effort: 

o Identifies best practices and moves forward from that point, 

o Looks at a broad range of materials on the subject and questions that are 
stated in the evaluation purpose. 

o Reviews existing work for the selected subject, as a preliminary step to the 
development of evaluation plans, 

o Capitalizes on professional evaluation standards, 

o Applies what is understood about "change". 

Is Results-Oriented with goal of Improving Instructional Practice 

o Has the capacity to disaggregate to answer specific data questions related to 
program improvement, 

o Is oriented toward improving programs and instructional practices which result 
in attainment of improved student outcomes, 

o Looks at opportunity-to-learn standards, 

o Balances outcome assessment with process/rights, 

o Evaluates post-school short and long-term outcomes. 

o Includes judgements/decisions which are standards-referenced. 

Reflects Consensus on Outcomes and Inclusive of All Students 

o Includes outcomes/performance indicators on which there is stakeholder 
consensus, 

o Applies performance indicators and standards that are inclusive and 
applicable to all students, yet reflect the unique needs of students with 
disabilities. 

o Is based on and identifies high expectations for all students. 
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Contains Incentives and Supports 

o Considers incentives and supports (technical assistance) for data collection. 

o Uses scenarios to demonstrate the use of data and possible 
program changes (i.e. how education for students will improve; 
and how things will change for the better for classroom 
teachers, the principal). 

o Provides stakeholder education regarding evaluation concepts, design, 
implementation, and use of the evaluation system. 

o Provides statewide training on the evaluation process and expectations. 

Meets Multiple Levels of Need and Use 

o Produces data that have meaning and utility at all levels collected (federal, 
state, and local), and used for decision making. This includes effective 
marketing and dissemination strategies. 

o Provides continuity among components of the evaluation system across all 
levels (classroom, school district, state, and federal). 

o Provides a core set of data that can be aggregated across states. 

o Provides for evaluation data and interpretations to be disseminated to and 
potentially used by all stakeholders. 

o Provides information that can be used to predict trends over time. 
Involves Stakeholders 

o Provides for evaluation data and interpretation to be disseminated to and 
potentially used by all stakeholders. 

o Is developed by a broad-based constituency that represents the diversity of 
persons in the community, including persons with a disability. 

o Has a process for broad-based stakeholders to be continually involved; 1) .up 
front, to decide what to evaluate and how to evaluate; 2). during evaluation 
to monitor the process and help figure out if it is going well and 3). al 
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periodic benchmarks to evaluate how it worked and the extent to which it 
resulted in improved outcomes/benefits for students and systems. 

Addresses Accountability Dilemmas 

o Includes rewards and sanctions to address accountability expectations of 
multiple clients (OSEP, SEA, state and local school boards, legislature, public, 
etc.). 

o Does not confuse compliance monitoring with evaluations recognizing that 
these are two different paradigms. 

linked to Reforms for All Students 

o Is tied to local school building improvement plans for all children, including 
those placed outside their home school. 

o Is linked to GOALS 2000 and Schooi-te-Work implementation, and other 
pertinent reform/restructuring efforts within each state. 

o Assesses the effectiveness of special education services as part of the 
assessment of all education services. 

o Ensures that all evaluation plans, efforts, and guidelines are in the context of 
state and local level reform. 

After a review of the work of day one, the remainder of the second day of the forum 
was spent by discussing the following questions: 

What will the state education agencies need to facilitate their planning and 
implementation of statewide evaluation systems (e.g., resources, assistance and policy 
changes at the federal, state, and local levels)? 

Following is a summary of the recommendations to OSEP and state education agencies that 
were generated by small and large group discussion. 

OSEP 

o OSEP should clearly communicate the importance of program evaluation as 
part of the National Agenda, and include the expectation of program 
evaluation within funding priorities, within requirements for state plans and 
in content within workshops and conferences, etc. 
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OSEP should develop and support systems for additional training that is 
needed at all levels (federal, state, and local) in the area of evaluation,, 

The OSEP plan should begin meeting the training needs of its staff including 
state contacts for Part B and other contract/project staff who will deliver 
evaluation technical assistance (e.g., Parent Training and Information Centers; 
Regional Resource Centers; and other technical assistance, training, and 
information projects). This could be a Division of Personnel Preparation 
priority. 

OSEP should conduct a needs assessment of its federal office and selected 
contract/project staff to determine the extent of prior knowledge thr.i staff 
have in the area of evaluation and then set forth plans io initiate a trainer of 
trainers model across divisions. This plan should include persons from all of 
OSEP divisions. 

OSEP should consider recommendations made by an earlier SAFES 
Workgroup, including extending the length of funded projects, linking the 
Regional Resource Centers to funded projects, and providing support for the 
development of evaluation designs (feasibility studies), 

OSEP should work toward bringing together general and special education 
policy implementers, who are working with GOALS 2000, to consider the 
work of this forum; (e,g,, National Center on Educational Outcomes, SAFES 
project personnel, Regional Resource Center staff, representatives of the 
National Academy of Science, NECTAS, National Association of State 
Directors of Special Education, etc). 

With the assistance of professionals from the area of educational evaluation, 
OSEP should include in its annual leadership meeting sessions on statewide 
program evaluation or devote an entire leadership meeting on statewide 
program evaluation, OSEP program staff whose training and experience are 
in program evaluation should be involved in planning and implementing these 
sessions. 

OSEP should convene other groups with general and special education 
counterparts responsible for program evaluation at the state level 

OSEP should conceptualize a congruent structure of statewide program 
evaluation in which special education program evaluation is integrated into 
overall education evaluation. 
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o OSEP should add to the evaluation criteria requirement of discretionary 
projects/applications the requirement to include a link to the overall 
statewide evaluation plan. 

o OSEP should place program evaluation assistance to state education agencies 
as a priority by Regional Resource Centers after and as an extension of the 
technical assistance "training of trainers" plan. 

o OSEFs workgroups regarding the re-authorization of IDEA should develop 
clear language that requires the inclusion of students with disabilities within 
GOALS 2000 policies, programs, procedures, and practice. 

o OSEP should set a priority for its re-authorization workgroup that IDEA 
should be reframcd to reflect a results-oriented, accountability system with 
child protections maintained. There should be incentives for states to develop 
and implement results-oriented supports and services at the local level that 
demonstrate high levels of learning for students with disabilities. 

o OSEP should seek assistance from professionals in the area of educational 
evaluation in defining compliance monitoring as a component that supports 
a results-oriented accountability system. A discrepancy analysis should be 
performed to identify chose regulations that support or create barriers to a 
results-oriented accountability system as part of the comprehensive 
educational system. 

o OSEFs workgroup for re-authorization should develop language to be added 
to IDEA that re-states the importance and value of statewide program 
evaluation. 

o OSEP should provide systems change grants/assistance for all state education 
agencies in the area of statewide program evaluation within an existing 
discretionary program or a new discretionary program created within IDEA 
re-authorization. 

o OSEP should organize a task force of educators from its staff, RRC, and the 
state level to collaborate regarding the structure of the IEP to support/reflect 
a results-oriented approach for each child. 

o OSEFs grants/applications for staff development at both the preservice and 
inservice training levels should require that proposals reflect a service 
orientation and that it should be is delivered through collaborative training 
approaches of general and special education. More funding, more 
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demonstrations, and coordination of funding for program evaluation are 
needed. 



OSEP should lead a movement aimed at determining how legislation and 
policies at the federal level can be implemented or revised to support 
collaboration across agencies and divisions within agencies. 

OSEP should articulate clear expectation about evaluation to SEAS, hold 
them accountable for the implementation of statewide program evaluation, 
ana give them the autonomy to do so within an established set of standards. 
The standards should be including in the re-authorization of IDEA (e.g., 
identify required core elements of statewide program evaluation). Beyond 
this set of expectations, SEAs should have autonomy in the planning and 
implementation process. 



State Education Agencies (SEAs) 



SEAs need to prioritize statewide program evaluation and clearly and 
consistently communicate the importance of evaluation. 

SEAs need to implement a congruent structure for program evaluation that 
integrates special education program evaluation within overall educational 
program evaluation. 

SEAs need to include the development and implementation of statewide 
program evaluation within SEA funding priorities (20% state discretionary 
funds). 

SEAs should provide guidance materials and training for local education 
agencies related to statewide program evaluation. 

It is important for SEAs to be aware of the current stress in the system; e.g., 
school districts are implementing a number of federal and state education 
reform activities and initiatives. 

State legislation as well as policies at the state and local levels should support 
collaboration across agencies and divisions within agencies. 
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IIL SUMMARY AND NEXT STEPS 



Several materials were utilized to stimulate discussion by forum participants: 
background documents pertaining to program evaluation sent prior to the forum, additional 
information included within forum packets, and a "strawman" of proposed set of parameters 
for a statewide system for evaluating programs and services for students with disabilities. 
Fifty components/ descriptors of a statewide program evaluation were identified in this 
forum within eleven major categories. In addition, a number of recommendations were 
made regarding resources, policy changes, and other assistance needed at the federal, state, 
and local levels to facilitate the planning and implementation of statewide systems to 
evaluate programs and services for students with disabilities. 

These recommendations included the need for clear communication at the federal 
and state level regarding the importance of program evaluation. There is also a need for 
training in the area of program evaluation to be provided for OSEP staff, SEA staff, OSEP- 
funded entities charged with providing technical assistance, and local school districts. Forum 
participants recommended that a congruent structure of program evaluation should be 
articulated at the federal and state levels in which special education program evaluation is 
integrated into overall educational evaluation systems. During the re-authorization process, 
forum participants recommended that changes be made to IDEA which clearly express the 
importance and value of statewide program evaluation and the focus on program and 
student results. 

It is important that OSEP and the SEAs place a higher priority on statewide program 
evaluation. Forum participants recommended that the OSEP annual leadership conference 
contain sessions on program evaluation. In addition, other workshops and conferences with 
general and special education personnel responsible for program evaluation should be held 
in order to facilitate joint planning and implementation within the states. 
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Invited Guests: 



Richard Baldwin 

Special Education Services 

Michigan Department of Education 

P.O. Box 30008 

Lansing MI 48909-7508 

517/373-9433 

Marsha Brauen 
WESTAT Corporation 
1650 Research Boulevard 
Rockville, MD 20850-3129 
301/738-3668 

Martha Brooks 

Division for Exceptional Children 
Department of Public Instruction 
P.O. Box 1402 
Dover, DE 19903-1402 
302/739-5471 

Sharon Freagon 

Department of Special Education 
EPCSE 

Northern Illinois University 
DeKalb, IL 60115 
815/753-0656 

Pat Gonzalez 
NSTEP at NASDSE 
1800 Diagonal Road 
Suite 320 

Alexandria, VA 22314 
202/519-3800 



Mary Ann Lachat 

Center for Resource Management 

2 Highland Road 

South Hampton, NH 03827-3607 

603/394-7040 

Nancy LaCount 

Kentucky Department of Education 
Div. of Exceptional Children Svs. 
Capitol Plaza Tower, 17th Floor 
Frankfort, KY 40601 
502/564-2672 

Jim Leinen 

Western Regional Resource Center 
1268 University of Oregon 
Eugene, OR 97403-1268 
503/346-5641 

Margaret J. McLaughlin 
Institute for the Study of 

Exceptional Children 
College of Education 
University of Maryland 
1302 Benjamin Building 
College Park, MD 20742-1161 
301/405-6495 

Ken Olsen 

MidSouth Regional Resource Center 

University of Kentucky 

126 Mineral Industries Building 

Lexington, KY 40506-0051 

606/257-4921 
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Gordon A. Schuetz 
Exceptional Education 

and Supportive Services 
5225 West Vliet Street 
Milwaukee, WI 53208 
414/475-8745 

Ed Wilkins 

Northeast Regional Resource Center 
Trinity College of Vermont 
208 Colchester Avenue 
Burlington, VT 05401 
802/658-5036 

Martha Williams 

Center for Resource Management 

2 Highland Road 

South Hampton, NH 03827-3507 

603/394-7040 

Dee Spinkston 

Federation for Families with Special 

Needs 
Suite 104 

95 Berkeley Street 
Boston, MA 02116 
617/482-2915 

James Ysseldyke 

National Center on Educational 

Outcomes 
University of Minnesota 
350 Elliot Hall 
75 East River Road 
Minneapolis, MN 55455 
612/626-1530 



U.S. Department of Education: 

Thomas Hehir 
Lou Danielson 
Gerrie Hawkins 
Jane Case Williams 
Ruth Ryder 



NASDSE Staff: 

Joy Hicks 
Eileen Ahearn 
Pecolia Freeman 
Joy Markowitz 
Judy Schrag 
Lyn Sweetapple 
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STATEWIDE EVALUATION OF PROGRAMS AND SERVICES 

FOR STUDENTS WITH DISABILITY 

August 30 th and 31*. 1994 
AGENDA 

Tuesday, August 30, 1994 

8:00 - 9:00 Continental Breakfast 

9:00 - 9:15 Welcome 

Joy Hicks - Director, Project FORUM 

9:15 - 9:30 o Opening Remarks/Background 

Review of Federal Program Evaluation Requirements and 
Other Considerations Impacting Program Evaluation 

Lou Danielson - Director, Division of Innovation and 
Development 

9:30 - 9:45 Participant Introductions 

9:45 - 10:00 Orientation and Logistics 

Judy Schrag - Policy Analyst, Project FORUM 
Martha Williams - Forum Facilitator 

10:10 - 10:15 ♦ ♦ ♦ Break ♦ ♦ ♦ 

10:15 - 11:45 Panel Presentation: Alternative Approaches Being Used by States to 

Implement Program Evaluation 

Pat Gonzalez - Information Services Coordinator, 
NASDSE - Panel Moderato, 

Jim Ysseldyke - Director, National Center on Educational 
Outcomes 

Mary Ann Lachat - President, Center for Resource 
Management 

Nancy LaCount - Branch Manager, Kentucky SEA 
Richard Baldwin - State Director of Special Education, 
Michigan SEA 

11:45 - 12:00 Overview of the Afternoon Work 

Martha Williams - Forum Facilitator 
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12:00 - 1:00 Lunch 

1:00 - 2:00 Suggested Components/Parameters of a Statewide Program Evaluation 

System - Programs and Services for Students With Disabilities 

Ken Olsen - Technical Assistance Specialist, Mid-South 
Regional Resource Center 

2:00 - 2:15 Comments - Evaluation of Programs and Services for Student With 

Disabilities 

Tom Hehir • Director, OSEP 

2:15 - 3:30 Discussion of Suggested Components/Parameters of a Statewide 

Program Evaluation System: Identification of Further Considerations 
for Re^oncepraalizing/Conceptualizing State Systems for Program 
Evaluation 

3:30 - 3:45 Break 

3:45 - 4:45 Small Group Sharing/Group Discussion: Components, Parameters, 

and Considerations: State Systems for Program Evaluation 

4:45 - 5:00 Summary of the Day 

Martha Williams - Facilitator 

Evening Optional Group Dinner 

Wednesday. August 31. 1994 

8:00 - 8:30 Continental Breakfast 

8:30 - 9:00 Plans/Expectations for the Day 

Martha Williams - Facilitator 

9:00 - 9:45 Challenges/Opportunities to Implementation of Program Evaluation 

9:45 - 10:15 Small Group Presentations/Group Discussion: 

Challenges/Opportunities to Implementation of Program Evaluation 

10:15 - 10:30 Break/Checkout 
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10:30 - 10:45 Existing Resources to Assist SEAs in Planning and Implementing 

Statewide Systems or Procedures for Evaluation of Programs and 
Services for Students With Disabilities 

Judy Schrag - Policy Analyst, Project FORUM 

10:45 - 11:30 Identification of Additional Federal/State/Local Resources, Policy 

Changes, and Other Assistance Needed to Support the Planning and 
Implementation of State Systems or Procedures for Evaluation of 
Programs and Services for Students With Disabilities 

1 1:30 - 12:00 Group Discussion of Existing and Needed Evaluation Resources, 

Policy Changes and Other Needed Assistance 

12:00 - 12:15 Summary/Next Steps 

Martha Williams - Forum Facilitator 
Joy Hicks - Director, Project FORUM 
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Have We Made Progress in Fifteen Years of Evaluating the Effectiveness of Spedal Education 
Programs? 



Introduction 

The desire of special educators to move beyond compliance monitoring to 
evaluating the effectiveness of special education programs has been expressed for 
over fifteen years (e.g., see Olsen, 1979). Approaches to evaluating effectiveness 
have been put forth since shortly after the passage of the Education of All 
Handicapped Children Act of 1975 (now the Individuals with Disabilities Education 
Act or IDEA) while noting the lack of consistent definition of terms, lack of 
consensus on a conceptual model to interrelate the inputs, processes and outcomes 
in special education programs and, most importantly, lack of agreement on what it 
means to have an effective special education program (Borich, 1977). While all of 
these concerns still remain today, a number of significant changes are taking place in 
the ways that evaluation of services to students with disabilities is viewed and in 
the ways evaluation is approached. The purpose of this article is to provide 
evaluators of such services with a context for their work; to help them understand 
how we have arrived at this point and, perhaps, what is on the horizon. 

Five strands of change can be identified that have implications for the present 
and future of special education program evaluation: 

From To 

Schools and districts as locus of State and Federal structures driving 

control, with limited Federal evaluation with more active Federal 

involvement. involvement. 

Evaluation for program Evaluation for accountability, 

improvement. 

Evaluation of inputs and processes. Evaluation of outcomes and effects 

in the context of inputs and 
processes. 

Simple models of inputs, processes More sophisticated conceptual 
and outputs. models taking in a larger number of 

factors. 

Student learning outcomes specific to Integration of outcomes in larger 
special education. general education and human 

service frameworks. 
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Local to State and Federal Control 

While most of the efforts of the 70's and '80's focused on helping schools and 
local education agencies (LEAs) plan and conduct evaluation (Olsen, 1984, 1986), 
there now is an increasing tendency for aggregate evaluations at state and Federal 
levels. State and Federal support for special education program evaluation has 
reflected this change. In the 1970's the U.S. Office of Special Education funded the 
Evaluation Training Consortium at Western Michigan University and the 
University of Virginia. Although the initial intent of this project was to develop 
program evaluation procedures for higher education, its work expanded to help 
guide LEAs in conducting self-evaluations,. The project culminated with the 
production of Program Evaluation: A Practitioner's Guide for Trainers and 
Educators (Brinkerhoff, Brethower, Muchyj, and Nowakowski, 1983). This 
evaluation reference tool for special educators spawned a number of state resource 
documents and training events. Workbook-type manuals and team training events 
in the states of Florida, Maine, Missouri, Kansas, Nebraska, Maryland, Virginia, 
West Virginia and Utah have roots in that guide, and, to a great extent, in each 
other. For example, the Florida materials (Florida Department of Education, 1985) 
were modified for use in Maryland and those materials were, in turn, modified for 
use in Virginia and, finally, West Virginia (McLaughlin, 1990b). Similarly, The 
Missouri Special Education Evaluation System (SEES) (Missouri Section of Special 
Education, 1985) led to the Utah SEES (Utah Consortium of Local Directors of Special 
Education, 1988). The intent of each was to provide step-by-step guidance to help 
LEAs plan and conduct their own evaluations. 

In 1988 the Council of Administrators of Special Education (CASE) produced 
Special Education Program Evaluation: An Overview, (McLaughlin, 1988) which 
drew from earlier efforts to provide a workbook-type approach for local evaluations. 
West Virginia and Kansas later expanded on the CASE document and used their 
manuals to conduct training and to provide follow-up technical assistance for local 
teams of special educators from schools and school districts (McLaughlin, 1990a). In 
all cases the evaluations were driven by locally identified needs and designed by 
local stakeholders. 

During this same era, other states were providing local systems with general 
handbooks on evaluating special education programs (e.g., California, 1982; Illinois, 
1982; Rhode Island, 1984). Other states were providing their LEAs with specific 
forms, data collection procedures, and report formats (e.g., Massachusetts, 1981; 
North Carolina, 1983; Connecticut - Maher, Coppola, Ongley and Kahn, 1986). 
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However, local school districts were given the option of using those tools or others 
that they chose. Some commercial materials on special education evaluation were 
produced (Nance and Borich, 1986; Maher and Bennett, 1984), but these too 
presented methods for local education agencies to select components of their special 
education system and design evaluations for those components. In addition to 
providing materials, training and technical assistance, states such as Maryland, 
North Carolina and others provided incentive funding for LEA-initiated self studies 
(Olsen, 1986). States, therefore, have provided support for LEA evaluations but 
historically have not required use of common formats and procedures. 
However, DeSrofano (1990) points out that: 

"The absence of specific guidelines or monitoring agency to standardize the 
types of evaluation to be collected permits creativity in evaluation design ... 
but exacerbates problems associated with cross-site aggregation of evaluation 
data. Use of different instruments across projects to measure student and 
program characteristics make combination or comparison of data problematic 
.... a system for standardizing the collection of certain evaluation data may be 
desirable to enable the meaningful aggregation of data across sites." (p. 262). 
Therefore, an increasing number of states are developing special education outcome 
and inoJcator systems that cut across LEA lines (e.g., Michigan, New Jersey) and 
nearly all states are looking at ways to move some of their state resources from 
monitoring LEAs for compliance to monitoring them from effectiveness- Over half 
of the states are exploring results-based or outcome-based monitoring either as an 
alternative or adjunct to its system of monitoring for compliance to Federal and 
state requirements. For example, Texas has been piloting a results-based monitoring 
system for two years (Gray, 1994). The Texas system is implemented by a state team 
that makes judgments about effectiveness as well as compliance. The effectiveness 
component is optional for school districts and currently carries no potential for 
sanctions. However, this approach is indicative of the tendency for states to take a 
more active role in structuring and coordinating evaluation activities. 

Until very recently Federal involvement in special education evaluation has 
paralleled state approaches, i.e., supporting local option evaluations and allowing 
states maximum flexibility in interpreting Federal requirements. In 1983 the Office 
of Special Education Programs (OSEP) funded an institute "to offer technical 
assistance and logistical support to the Council of Administrators of Special 
Education (CASE) and to as many as 12 Local Education Agencies (LEAs) in the 
implementation and testing of models for the evaluation of special education 
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programs" (CASE, 1983). This project and most subsequent efforts by OSEP 
demonstrated that the Federal government had little interest in mandating special 
education evaluation standards or practices nor in gathering national data on 
effectiveness. 

Evaluating the effectiveness of special education and specifically of 
individualized education programs as required by IDEA has proceeded without 
Federal direction. For example, there has always been a requirement in state IDEA 
plans for a section describing how the state will conduct evaluation of special 
education program effectiveness and specifically how the state will evaluate the 
effectiveness of individualized education programs (IEPs). Federal attention to this 
requirement has been spotty, at best. The National Association of State Directors of 
Special Education (NASDSE) recently analyzed the evaluation components of state 
plans over time and found: 

• An increase in the number of states that discuss compliance monitoring in the 
annual evaluation sections of the State Plan and Part B Performance Report. 

• An increase in the number of states that describe a two-tiered approach to the 
annual evaluation requirement, whereby the LEAs have specific evaluation 
requirements and the state has specific monitoring requirements. 

• An increase in the number of states that encourage LEA "self-study." This 
practice was described by different states as a pre-monitoring tool or an 
evaluation tool. 

• An increase in the number of states that provide technical assistance to LEAs to 
help them evaluate their programs and procedures. (Gonzalez, 1992). 

Such plans were approved by OSEP, and therefore the Federal government tacitly 
endorsed evaluation as something that was locally designed and driven. Attempts 
were made by OSEP and its contractors in 1985, 1987 and 1990 to produce more clear 
criteria for these sections of the state plans (personal communications), but each 
attempt failed to be approved as an official guideline for state evaluation systems. 

However, the winds might be changing. Perhaps the most aggressive Federal 
effort to date has been the State Agency /Federal Evaluation Studies (SAFES) 
program. The SAFES program provides for cooperative agreements between the 
U.S. Department of Education and state agencies to conduct evaluations of special 
education programs and services. Through this program, states propose evaluation 
studies and the Federal government provides funding and technical assistance for 
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the state-designed efforts. The program has evolved from a series of unrelated and 
unguided studies to an increasing emphasis on themes that are of national concern 
(e.g., assessment of outcomes, especially as they relate to national frameworks, has 
been increasingly evident in funding announcement priorities). 

The funding of the National Longitudinal Transition Study (Wagner, 1990) 
and the National Center on Educational Outcomes (NCEO) at the University of 
Minnesota are evidence of an increased Federal interest in more commonalty in 
state and national evaluations of services to students with disabilities. In addition, 
OSEP appears to be placing more emphasis on projects with national data collection 
potential, such as the Performance Assessment for Self-Sufficiency -Project PASS 
(Campeau, 1993). 

Therefore, both state and national support for evaluation of services to 
students with disabilities appears to be moving toward centralized information 
collection and away from LEA-driven evaluations. Olsen and Massanari (1991) 
have argued for a balanced approach that includes having some ongoing data 
available at the state level on services to students with disabilities and on the effects 
of those services, a system for SEA self-evaluation, and support for LEA self- 
evaluations. It is likely that pressure toward more centralized evaluation functions 
will continue as the public continues to demand more data on overall cost and 
effects of services for students with disabilities. 

From Program Improvement to Accountability 

Paralleling the increasing emphasis on centralized evaluation is a change in 
the purpose for which special education evaluations are conducted. The clearly 
defined purposes of the past: to improve the services and programs; are giving way 
to a new purpose: accountability. Horvath (1985) defined evaluation for program 
improvement as a management tool for identifying specific components of the LEA 
special education program that need attention. He defined evaluation for 
accountability as a way of providing information to administrative, regulatory, 
oversight or funding authorities about the operations and effects of a program. 

The change from program improvement to accountability as the purpose for 
special education program evaluation is consistent with the national demands for 
reform in general education. No longer are stakeholders willing to leave 
educational decision making in the hands of educators. Persons with a stake in the 
education of students with disabilities are asking if special education services are 
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resulting in expected changes and even if the effort is worth the results that are 
being achieved (Lewis, 1991; Zirkel, 1990). 

"Outcome-based accountability" and "results-based monitoring" have become 
bywords. Even though an assumption of outcome-based accountability systems is 
that they will improve student performance, the emphasis of these new systems is 
on ensuring that the educational system delivers what is expected of it (Center for 
Policy Options in Special Education, 1992). Stakeholders are demanding 
information on the extent to which expected results are being obtained and are 
demanding that rewards and sanctions be established as contingencies for 
accomplishment of outcomes. Even the National Agenda for Achieving Better 
Results for Children and Youth with Disabilities recommends that special educators 
"(i)mplement accountability systems that monitor program effectiveness, balancing 
process with results, and provide incentives for program improvement and 
sanctions for noncompliance. " (COSMOS, 1994, p. 18). 

The potential negative side effects of using incentives and sanctions in a 
rigorous accountability approach to program evaluation in special education have 
yet to be fully explored (Olsen, 1994; Center for Policy Options in Special Education, 
1992). While the debate among assessment and evaluation researchers continues to 
escalate regarding the feasibility of using the same assessment systems for both 
program accountability and improvement purposes, policymakers are proceeding as 
if there will be few problems. However, it is becoming increasingly apparent that 
high stakes environments can lead to superficial adherence to standards, 
inappropriate placements and restricted curricula and instruction (Rogers, 1983; 
Fullan, 1991; Center for Policy Options in Special Education, 1992). 

From Evaluation of Process to Outcomes and Back Again 

Evaluation of special education programs and services has historically 
focused on answering such input questions as, "Are staff qualified?", "Are materials 
available?", and "Are facilities accessible?" or such process questions as, "Were 
parents involved in the decision process?", "Does the student's program match his 
IEP?", and "Are students participating with their nondisabled peers?" However, 
the growing dissatisfaction with access and process as the intent of special education 
and the increasing demand for more information on outcomes is well documented 
(Olsen, 1979; Borich and Nance, 1987; Vogelsberg, 1994; George, George and 
Grosenick, 1990). Now there is a new realization - that outcomes data without 
information on the programs and services that led to those outcomes will leave the 
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public with no basis for making decisions about needed changes (Ysseldyke, 
Thurlow, Bruininks, Gilman, Deno, McGrew, and Shriner, 1992). 

The Annual Reports to Congress from OSEP have provided information on 
the numbers of students in special education, on the number of teachers available 
and needed and on the extent to which students are in integrated placements, but, 
until recently, have provided very little information on the outcomes of those 
inputs and processes (DeStefanb and Wagner, 1990)). A few special projects such as 
the National Longitudinal Transition Study (Wagner, 1990) have provided the 
nation with a limited picture of such outcomes, but such information has been 
scarce. The Robert Wood Johnson Foundation special report entitled "Serving 
Handicapped Children" (1988) stated that the "emphasis on the availability and use 
of services should not obscure concerns about the quality of these services, concerns 
that cannot be addressed without substantially more information about children's 
progress over time" (p. 17). Likewise, the report of the National Council in 
Disability (1989) described the feelings of parents, educators and taxpayers in every 
locale: 

The time has come to ask the same questions for students with disabilities 
that we have been asking about students without disabilities: 

- Are they achieving? 

- Are they staying in school? 

- Are they prepared to. enter the work force when they finish school? 

- Are they going on to participate in postsecondary education and training? 

- Are they prepared for adult life? (p. 2) 

In response, the National Agenda for Achieving Better Results for Children and 
Youth with Disabilities (COSMOS, 1994) recommends that agencies define program 
results and functional learning outcomes for children and youth with disabilities 
and develop indicators that focus on those results and outcomes at the local, state, 
and building levels in collaboration with children and youth, families and 
community members. States such as Michigan, Delaware and Kentucky have made 
concerted efforts to produce disaggregated data on the achievement levels of 
students with disabilities. The NCEO has worked with the leaders of the National 
Assessment of Educational Progress to increase the number of students with 
disabilities included in that measure, with the intent of increasing the pool of 
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information available on outcomes (as well as improving the comparability of 
results across states) (Ysseldyke, Thurlow, McGrew and Vanderwood, 1994). 

However, a number of authors have pointed out that information on 
outcomes, especially negative outcomes, provides little information about what 
actions to take to improve those outcomes (Horvath, 1992; Ysseldyke, Thurlow, 
Bruininks, Gilman, Deno, McGrew, and Shriner, 1992; Olsen and Massanari, 1991). 
While information on outcomes is necessary, it is not sufficient. There is growing 
awareness that it is essential to also have information on the inputs and processes 
that lead to achievement of those outcomes. The national emphasis on 
M Opportunity-to-Learn Standards" in Goals 2000; Educate America Act (1994) is 
evidence that the pendulum is swinging back to the middle. Opportunity-to-learn 
standards have yet to be defined, but most likely will be drawn from the effective 
schools literature (Porter, 1993). Porter suggests that such standards can be used to 
explain outcomes (and he recommends using them for that purpose rather than as 
an accountability measure). 

Future efforts to evaluate educational services to students with disabilities 
must recognize the need to gather information at all levels of implementation in 
order to understand what outcomes are or are not being achieved and what 
adjustments must be made to improve those outcomes. Stakeholders who have a 
vested interest in some of the process provisions of IDEA might want to advocate 
for folding key provisions into local and state Opportunity-To-Learn standards. 

From Simple to Complex Conceptual Models 

Chen (1990) suggests that evaluation is most effective when it is informed by 
a conceptual model of the program being evaluated. Such a model, frequently 
presented in both graphic and narrative form, defines the components ot a program 
design and the interrelationships among those components (Lynch and Covert, 
1981). Models used to drive special education evaluation have evolved from simple 
input-process-output lists to complex models which take into consideration context 
factors and external influences to special education services. The earliest models 
tended to be based on lists of activities or inputs, resources or processes, and 
outcomes or results that were considered to reflect an effective program, without 
demonstrating interrelationships among the components or taking into account 
factors outside of a narrow special education focus (e.g., see Olsen, 1979; Lieberman 
and McNeil, 1982; Borich, 1977; Gable, 1982; and Maher and Bennett, 1983). 
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More recently, models for services to students with disabilities have started to 
take into account a greater number of variables. The New Hampshire special 
education program improvement partnership incorporates a multi-level model and 
a wide range of indicators in the areas of 
« student and program outcomes; 

• philosophy, policies and procedures; 

• resources; the school program; 

• instructional practices; 

• staff competencies, attitudes, and relationships; 

• parent participation; 

• school and classroom climate; and 

• leadership (Lachat, Williams, and Brody, 1986). 

West Virginia developed a multi-level model that defined inputs, processes and 
outcomes in ten activity areas: 

• parent and community support, 

• school based assistance teams, 

• identification and referral, 

• multidisciplinary assessment, 

• eligibility determination, 

• development of the IEP, 

• placement and implementation, 

• review of the IEP, 

• exiting and transition, and 

• program administration. 

Outcomes from each activity are shown to be inputs for other activities and the 
interrelationships among effectiveness indicators is shown graphically and in 
narrative form (Olsen and Turley, 1988). 

DeStefano and Wagner (1990) indicate that a conceptual framework "should 
include both proximal and distal outcomes, key independent variables that are 
expected to influence outcomes and indications of the expected relationship among 
them" (p. 21). Their model for outcome assessment in special education 
encompasses not only school programs and services and student outcomes, but also 
school contexts, individual /family/ community characteristics, young adult 
outcomes and adult programs /services. 
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Education, especially education of students with disabilities, can no longer be 
viewed as a simple factory model of inputs, processes and outputs. Influences 
include context factors, family demographics, and a variety of interventions by other 
health and human service providers. In the future, persons evaluating special 
education programs will have to build into their conceptual models an increasing 
number of factors as human services become more interagency in nature and as 
educational services to students with disabilities become increasingly imbedded in 
services for all students. 

Effectiveness Indicators for Special Education to Effectiveness Indicators for All 

Effectiveness indicators in special education have been changing to reflect the 
changing service delivery system. Whereas indicators in the past described special 
education inputs, processes and outcomes in isolation, they are increasingly being 
described as a part of indicators for all students and for society in general. The 
effectiveness indicators (or standards, criteria, goals or targets) provide the basis for 
comparison to the actual state-of-affairs in a discrepancy evaluation model (Provus, 
1971). 

One of the earliest attempts to synthesize effectiveness indicators in special 
education was the work by the National RRC Panel on Indicators of Effectiveness in 
Special Education (1986). The Regional Resource Center (RRC) program conducted 
needs assessments in all SEAs and found that over 60% of the states had a need for a 
comprehensive reference tool by which agencies could judge their own 
effectiveness. An RRC task force found that "a great deal of redundant activity was 
occurring as state and local agencies sought to locate extant indicators of 
effectiveness in special education...the task force proposed that a document be 
developed...with the direct involvement of a broad-based panel of representative 
stakeholders" (p. 7-8). Research articles and state-developed indicator lists were 
synthesized into a comprehensive list of effectiveness indicators in six categories: 

• philosophy, 

• policies and practices; 

• resource allocation; 

• staffing and leadership; 

• parent participation and community involvement; 

• instruction; and 

• program and student outcomes. 
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There were hundreds of indicators in the first sections, but only a few in the 
program and student outcome section. The indicator lists were subsequently used as 
adjuncts to state evaluation systems in a number of states (e.g., Maryland, New 
Hampshire, Florida) and at least one state, West Virginia, added new literature on 
pre-referral processes and reformatted the indicators into inputs, processes and 
outputs according to their model (Olsen and Turley, 1988). While the document 
was linked to general education effectiveness indicators and designed with the 
general education audience in mind, it had a clear special education focus. 

A number of other indicator documents specific to special education were 
produced in the '80's and reported in the literature (e.g., Bickel and Bickel, 1986), in 
state reference guidelines (e.g., Florida, Virginia, Missouri, Utah, Kansas) and in 
local school district standards documents (e.g., Jefferson County Public Schools, 
1988). These references usually addressed special education programs in general. 
However, a variety of indicator documents also were developed specific to early 
childhood programs (e.g., Iowa's Early Childhood Special Education Review), 
secondary transition programs (e.g., Hasazi, Hock and Cravedi-Cheng, 1993), and 
especially for services to students with severe disabilities (e.g., University of 
Vermont, 1987; Kleinert, Smith and Hudson, 1990; Meyer, Eichinger and Park-Lee, 
1987; Johnson and Gadberry, 1981; Halvorsen and Sailor, 1989). As with the 
Effectiveness Indicators for Special Education Programs,, these documents focused 
on practices and resources rather than effects and outcomes. All of the documents 
were specific to special education, however. 

The national movement toward inclusion is leading to a different view of 
effectiveness indicators for educational services to students with disabilities. 
Perhaps the watershed documents were produced by Gartner and Lipsky (1989) and 
Stainback and Stainback (1984) as they called for a unitary system of education and 
for personnel specializing in services for students with disabilities to become 
support staff in a regular education environment. The National Association of 
State Directors of Special Education endorsed this concept and other special 
education and general education groups are in accord. For example, the National 
Agenda for Achieving Better Results for Children and Youth with Disabilities calls 
for "a unified educational system (which) will incorporate equitable standards and 
high expectations for all children and youth M (COSMOS, 1994, p. 3). 

General educators also are calling for a more inclusionary system. For 
example, the National Association of State Boards of Education endorsed such a 
system in its Winners All document (NASBE, 1992). Goals 2000 codifies the eight 
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national education goals and establishes a (NESIC). Likewise, the House and Senate 
committee language for Goals 2000 make it clear that Congress intend* for the 
National Educational Standards and Improvement Council to ensure that state and 
local standard setting and assessment activities provide for all children in relation to 
the same eight national education goals. 

The most recent effort to produce a comprehensive outcome and indicator 
system reflects this inclusionary orientation. The NCEO has developed a conceptual 
model of outcomes and is producing outcomes and indicators at six age levels (age 3, 
age 6, grade 4, grade 8, students exiting school and post school) in the areas of 

• presence/ participation, 

• accommodation/adaptation/ compensation, 

• independence/ responsibility, 

• physical /mental health, 

• social/behavior skills, 

• contributions / citizenship, 

• literacy and 

• satisfaction (Ysseldyke, Thurlow and Shriner, 1992). 

The entire model is based on an assumption of an integrated system. In fact, their 
conceptual model and all indicator documents are designed to apply to all students. 
Care has been taken to avoid language that implies a special education orientation 
(Ysseldyke, Thurlow, Bruininks, Gilman, Deno, McGrew, and Shriner, 1992). State 
personnel also are being very careful to avoid being seen as separatists, ensuring that 
functional curricula and unique special education goals are eliminated or at least 
cross-matched to the outcomes and indicators for all students. 

The debate regarding whether effectiveness indicators, and especially 
outcomes for all students, can meet the needs of students with disabilities is not 
over. Michigan has been engaged in a massive effort to identify the unique expected 
outcomes for students who exhibit specific impairments (e.g., visual impairment, 
hearing impairment, emotional impairment, learning disability). Expected 
outcome lists in Michigan usually begin with a statement such as, "Student 
completes the local minimum graduation requirements in all subject areas" (Frey, 
1991); however, the remaining outcomes are always unique to the type of 
disability/ impairment. The logic for separate outcomes has to do with the purpose 
of special education services. Michigan perceives that: 

"...the task of special education .. .is t o limit handicaps , to compensate for the 
disability or limit the inability, to enable individuals to access the same 
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knowledge as his (sic) nonhandicapped peers. To do this, special education 
attempts to discover what special education accommodations are necessary 
for students with an impairment to learn and achieve in education" (Frey, 
1991, p. 6, emphsais in original). 
The concern that the uniqueness of special education and its mission to provide 
specially designed instruction to students with disabilities will be lost is shared by 
others who fear that high expectations and common goals will mean that all 
students will have to meet the same standards in the same way regardless of 
disability, without accommodations (Little, 1993). Similarly, the disability 
community at large continues to have concerns about achieving positive outcomes 
unique to individuals with disabilities through all supports and services including 
residential, vocational, and social in addition to educational. The recently published 
Outcome Based Performance Measures by the Accreditation Council on Services for 
People with Disabilities (1993) specifies 30 life role outcome measures specific to 
people with disabilities and 17 outcome measures for organizations that provide 
supports and services for persons with disabilities. 

Developers of outcomes and effectiveness indicators must find ways to 
maintain a focus on what makes education effective for all students and ensure that 
students with disabilities are included in the overall picture of education. At the 
same time, they must ensure the availability of outcomes and indicators that 
provide a vision of effective practice and a basis for gathering information about 
how to better meet the needs of students with disabilities. Educators must be 
prepared to respond to the expectations of advocates and professionals for positive 
life role outcomes. Specifying both generic and specific outcomes may be the only 
way to address both needs. 

Summary and Conclusions 

Fifteen years of evaluation evolution in special education have resulted in 
both negative and positive changes. Evaluation of educational services to students 
with disabilities still lacks clear definitions of terms. Debates about purposes, 
standards and procedures abound. Clear guidance from the Federal level regarding 
requirements for evaluating effectiveness of individualized education programs is 
probably not forthcoming, but there is expanding interest in having common data to 
answer critical outcome questions. Unfortunately, the orientation at state and 
national levels seems to have shifted from a focus on program improvement to a 
focus on accountability without a full understanding of how evaluative 
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information might be affected by a high stakes environment. Evaluation of 
outcomes for students with disabilities and evaluation of the effects of supports and 
services to individuals with disabilities are now viewed in the broader framework 
of conceptual models that includes context, input and process variables that affect 
those outcomes and effects. 

The past fifteen years of experiences in evaluating services for students with 
disabilities lead this author to make the following recommendations for future 
evaluators: 

1. Approach the evaluation enterprise with a clear purpose in mind, determining 
whether the primary intent is to ensure accountability or program 
improvement, and being aware that one system is unlikely to meet both 
purposes. 

2. Advocate for development of multiple approaches at the state and national 
levels that support local level decision making without compromising state 
level data integrity. 

3. Use incentives and sanctions cautiously, being aware of the potential negative 
side effects of high stakes environments. 

4. Base evaluations on conceptual models that show the expected linkages among 
the inputs, processes and outcomes of each educational service and experience 
that affect students with disabilities in order to interpret and use the evaluative 
information. 

5. Define effectiveness indicators and especially outcomes for students with 
disabilities within the context of outcomes for all students, ensuring that the 
unique needs of students with disabilities for access, accommodation, 
adaptation and compensation are addressed. 

We can benefit from the developments and experiences of the past. Doing so could 
mean that another retrospective such as this in the year 2009 might describe a state- 
of-affairs in which adequate information on the progress of students with 
disabilities is being documented and used within a comprehensive evaluation 
framework for all of education. 
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